Plausibility of Multiframe Super Resolution For Reconstruction of DLP 3D Printer Projections.

Matthew Heydorn

Mentored By Dr. Greg Nordin

BYU Directed Research Jun 20, 2017

Abstract

An effective way to measure the quality of a DLP 3D printer is to examine the characteristics of the light projection produced by the printer's DLP array. Since the pixel size of the many DLP array 3D printers reach into the 10s of micrometers, it is expensive to obtain a camera with a sufficient pixel size to yield a useful image of the projections from the DLP array. To help solve this problem, this paper investigates the use of a multiple-frame super resolution algorithm to attempt to generate a high-resolution image from an array of offset low res images of the DLP's projection on an opaque surface. This paper will show that accurate reconstruction is only possible for images that do not contain high frequency features and will characterize the symptoms of attempting to reconstruct an image with high frequency features.

Approach

Algorithm

The algorithm used is from a paper written by Michal Irani and Shmuel Peleg[1] shown in Figure 1.

 

Figure 1: Super Resolution Algorithm

The algorithm takes an iterative approach. f(n)(x) is the nth estimate image of the initial guess (which is created by averaging all the initial low res images). The process involves two sets of low res images which are (1) the original low res images (gk) and (2) an iterative set of low res images created by down sampling the iterative estimate (f(n)(x)) to the low-res pixel space (gk(y)). Each image in original low res image set (gk) is subtracted to from the image with the corresponding offset in the iterative set of low res images. The result of each subtraction is added together to form a single diff image, which is then added to the previous estimated image to form the next estimated image.  The rightmost term to correct for imperfections in the camera's optical system. However, since our camera has no optical system (it has no lens) this term can be replaced with 1/c, with c being an arbitrary constant that will define how quickly the iterative estimates converge to a high-res image.

Calculating Offsets

For this algorithm to work, each image must be assigned a corresponding offset. The dataset we use was created by mounting a camera with a 4.4 micrometer pixel size to a piezo actuator capable of moving in 0.1 micrometer increments. Although we annotated the position of each image when they were taken, the reliability of using the spatial offsets as the offsets to feed into the algorithm was poor.

To solve this, we cropped each low res image around a single isolated point feature and calculated the center of mass for the feature in each low res image. We then selected the image with the lowest x and y center of mass and assigned it offset (0,0). We then calculated all the other image offsets relative to it. A low-res image with its center of mass marked by this process is shown in Figure 2.

 

Figure 2: One of the 179 lowres images used with its center of mass marked by the intersection of the two blue lines. This center of mass was then used to assign the image an offset.

By doing this we discovered that the offsets were very bi-modal. Of the original 255 low-res images available in the data-set, 76 of them clustered around one point and the other 179 around another point. To solve this, we had to discard the 76 images and only use the remaining 179 low-res images as input. After doing this we ended up with the offset distribution shown in Figure 3.

 

Figure 3: Offset distribution of the 179 images in our data-set.

Testing

Testing was done by offsetting a known high -res image using the same offsets shown in the distribution in Figure 3 and then down sampling each offsetted image. This provided us with a low-res set of images whose ideal convergence was known. Data gathered this way will be referred to as generated data. Data created from the actual low res offsets will be referred to as the real data.

Results

Running the Algorithm on Smooth Generated Data

We ran the algorithm on a variety of generated data-sets with a variety of offsets distributions and parameters to the algorithm.  Figure 5 shows the convergence behavior of a smooth Gaussian curve meant to represent the image produced by the DLP array. Figure 4 is the original high res image to which the algorithm should converge.

 

 

Figure 4: Generated image meant to characterize the projection created by four DLP pixels illuminated close together. This image was then offseted 179 times and each offset was down sampled by a 1:10 ratio to generated a set of low res images used as input to the algorithm. Note that this image contains few high frequency features.

 

Figure  5: Horizontal line scan at y=50. Note how the estimated image (green line) converges to the correct high-res output (red line).

 

Figure 6: (Left) The output of the algorithm after 100 iterations. Note the significant improvement from the guess (right) created by a trivial averaging of the low-res images.

The results are summarized in Figure 6. This type of algorithm is very effective for this type of image.  

Running the Algorithm on Generated Data With Sharp Corners

We generated an image similar to Figure 4 but with perfect 90 degree corners to see how the algorithm behaves around non-gaussian curves. For brevity, only the line scan is shown in Figure 7.

 

Figure 7: The algorithm has little trouble converging to a sharp corner. The spike at x=100 is caused by being near the edge of the image (which cannot be reconstructed due to the nature of the offsets)

Based on the convergence behavior in Figure 7, we can hypothesize that an image with sharp corners can be accurately reconstructed using the algorithm outlined in this paper, though with noticeable ringing before and after the corner areas.

Running the Algorithm on Generated Data With Sharp Corners in Close Proximity

Figure 8 shows the image we are attempt to reconstruct this time. Unlike previous images, Figure 8 has sharp corners which do not flatten out in between.

 

Figure 8: Image similar to Figure 4, except with higher frequency features (shorter in the spacial domain).

 

 

Figure 9: Line Scan Along Y=45 of Figure 8

Figure 9 demonstrates the weakness of this algorithm. Features that are both high amplitude and short spatially can not be accurately reconstructed. Furthermore, Figure 9 shows how such features generate noticeable dips in the estimated image (green) before and after  the two spikes. Figure 10 shows how these dips translate into a grid like effect superimposed on the estimated image.

 

Figure 10: Estimated Image after 100 iterations (left) and initial guess when trying to reconstruct the image in Figure 8 (right)

Running the Algorithm on the Real Data

 

Figure 11: Initial guess generated from the real DLP images (before running the algorithm) (left) and the same image in the frequency domain (right)

The Initial guess suggests that the DLP image has a single light source around the center and slopes down slightly sharper near the top left corner then toward the bottom right corner. However, when viewed in the frequency domain it appears that there is a large spike in the center of the image (shown by the vertical dark lines in right of Figure 11). This suggests that this reconstruction will have the same problem as the reconstruction of  the image in Figure 8.

 

Figure 12: Line scan at Y=45 of of the algorithm running on the real data. There is no red line since the ideal high-res image is not known.

Figure 12 seems to confirm our suspicions that this reconstruction will have the same problem as the reconstruction of the image in Figure 8. Comparing Figure 12 to Figure 9 shows a similar behavior, which leads to the conclusion that the real ideal high-res image contains a spike in the center somewhat similar in shape to the spikes in Figure 8.

Conclusion

We have shown that the super resolution algorithm proposed by Michal Irani and Shmuel Peleg [1] is only effective for reconstructing ideal high res images that do not contain distinctive high frequency features. Furthermore, we have shown what behavior this algorithm will exhibit when trying to converge to a high frequency feature. Since most DLP projections contain high frequency features, using such a super resolution algorithm (in the way outlined in this paper) is only partially effective for reconstructing DLP projections.

Future Work

Figure 9 shows how the algorithm dips down before a spike. It may be possible to generate information about the shape of the spike as a function of the dip that the algorithm makes before and after the spike, as well as how quickly the estimate line (green) rises.

Although this project involved running the algorithm tens of thousands of iteration on a variety of generated images, the conclusion here is not concrete, but rather a hypothesis to serve for further exploration.

Notes

[1] Michal Irani and Shmuel Peleg CVGIP: Graphical Models and Image Processing Vol. 53, No. 3, May, pp. 231-239, 1991 file: "1991 IRANI.pdf"

Appendix

Part of this project involved adding sub-pixel functionality to the original algorithm. This was done successfully, but made little difference to the functionality of the algorithm. Consequentially it is not described in detail in this paper. However, the basic premise involves replacing the integer shift function with one that calculates the sub pixel boundaries between pixels, then calculates each pixel value by the addition of four weighted areas per pixel determined by the sub-pixel boundaries.  Figure 13 outlines how this is done.

 

Figure 13: Sub Pixel Shift Function